Import, Clean, and Explore Dataset¶
Date Last Update: 2021-08-11
In [2]:
from pandas_profiling import ProfileReport
import pandas as pd
import numpy as np
from sklearn.preprocessing import MinMaxScaler
import datetime
import math
from matplotlib import pyplot as plt
In [3]:
# Load the data
#raw_data = pd.read_csv('/Users/lcags/Documents/Practicum2/DATA/max-entry-for-bb100-bb100-project-QueryResult_Reduced.csv')
raw_data = pd.read_csv('/Users/lcags/Documents/Practicum2/DATA/BB100_FullData.csv')
In [4]:
pprofile = ProfileReport(raw_data)
pprofile
Out[4]:
Data Exploration Notes¶
Univariate Notes
- weekid: Use to make seasons, drop in final. Use Year to make decade.
- week_position: Last position on chart before dropping, drop
- song: useful for data visualization, not for classification. Drop
- performer useful for data visualization, not for classification. Drop
- weeks_on_chart use to determine longevity. Median = 10, Q3 = 16.
- peak_position: Use to get top 10 hits.
- Spotify_Track_Duration: Missing values. Normal distribution.
- Spotify_Track_Explicit: Missing values. Skewed towards "False". Does not necessarily denote whether the song on the chart was explicit, this is just the entry found from Spotify API. Drop.
- Dancebility: Missing values. Normal looking distribution. Range between 0.1 and 0.9.
- Energy: Left skewed, songs tend to have more energy. Values between 0.1 and 0.9.
- Key: Missing values. Value between 0 and 10. Bimodal.
- Loudness: Slight left skew. Missing values. Value from -18 through -0.4.
- Mode: Missing values. Skews towards True. 60% of the values are true. Mode is major or minor key.
- Speechiness: Missing values. Extreme right skew. Values from 0.02 through .9
- Acoustiness: Missing values. Extreme right skew. Values from 0.000002 through .9
- Instrumentalness: Missing values. Large amounts of the values are 0. Drop.
- Liveness: Missing values. Right skew. 0.02 through .9
- Valence: Missing values. Slight left skew. 0.03 through .9
- Tempo: Missing values. Right skew, bimodal. 59 through 213. Rescale?
- Time Signature: Missing values. 95% of the values are 4/4 time signature. Drop.
Correlations
Energy is highly correlated with both loudness and acousticness.
Dancebility is highly correlated to tempo and time signature.
Weeks on Chart and Peak Position are positively correlated: The longer a song has been on the chart, the higher the peak the position.
Immediate Drops
- Danceability, Energy, Spotify_Track_Explicit, Week_Position
Missing Values
- According to missing value heatmap, if a song is missing one of the spotify values, it's missing all of them. All the missing values are correlated with each other. Drop only the rows that contain no Spotify data since it does not make sense to try to make up values for the entire entrie with chart data and no spotify data whatsoever.
In [99]:
raw_data = raw_data.dropna()
In [65]:
# go back to what i was doing before
raw_data = raw_data.drop(['danceability'
,'energy'
,'spotify_track_explicit'
,'week_position']
,axis=1)
In [66]:
raw_data.shape
Out[66]:
In [67]:
raw_data
Out[67]:
Add flags for peak position and long lasting¶
Top 10 is based on if the song peaked in the top 10 and longevity is based on the Q3 of the distribution of the datta.
In [51]:
raw_data['top_ten'] = np.where(raw_data['peak_position']<=10, 1, 0)
raw_data['long_lasting'] = np.where(raw_data['weeks_on_chart']>=5,1,0)
Add Seasons¶
In [52]:
#Function to convert week to seasons
def makeseason(week_id):
mdate = datetime.datetime.strptime(week_id, "%Y-%m-%d")
if mdate.month >= 12:
season = "winter"
elif mdate.month >= 9:
season = "fall"
elif mdate.month >= 6:
season = "summer"
elif mdate.month >= 3:
season= "spring"
else:
season = "winter"
return season
In [53]:
raw_data["season"] = raw_data["weekid"].apply(makeseason)
Add Decade¶
In [54]:
def make_decade(week_id):
mdate = datetime.datetime.strptime(week_id, "%Y-%m-%d")
decade = math.floor(mdate.year/10)*10
return(decade)
In [55]:
raw_data["decade"] = raw_data["weekid"].apply(make_decade)
Year¶
In [56]:
#make year delete if keeping with decade
def make_year(week_id):
mdate = datetime.datetime.strptime(week_id, "%Y-%m-%d")
return(mdate.year)
In [57]:
raw_data["year"] = raw_data["weekid"].apply(make_year)
Export Dataset Here¶
In [58]:
#For tableau viz
raw_data.to_csv(r'/Users/lcags/Documents/Practicum2/DATA/raw_data_visualization.csv'
,index = False
,header=True)
In [59]:
scaler = MinMaxScaler()
In [28]:
raw_data[['spotify_track_duration_ms']]=scaler.fit_transform(raw_data[['spotify_track_duration_ms']])
raw_data[['key']]=scaler.fit_transform(raw_data[['key']])
raw_data[['peak_position']]=scaler.fit_transform(raw_data[['peak_position']])
raw_data[['weeks_on_chart']]=scaler.fit_transform(raw_data[['weeks_on_chart']])
raw_data[['loudness']]=scaler.fit_transform(raw_data[['loudness']])
raw_data[['tempo']]=scaler.fit_transform(raw_data[['tempo']])
In [29]:
raw_data
Out[29]:
In [30]:
raw_data["year"].value_counts().sort_index().plot.box(vert = False)
Out[30]:
Drop incomplete years¶
In [31]:
raw_data.drop(raw_data[raw_data['year'] == 1958].index, inplace = True)
raw_data.drop(raw_data[raw_data['year'] == 2021].index, inplace = True)
In [32]:
raw_data.shape
Out[32]:
In [33]:
raw_data["year"].value_counts().sort_index()
Out[33]:
Dummy Coding¶
In [34]:
dummy = pd.get_dummies(raw_data
, prefix_sep='_'
, columns=['mode','season'])
In [35]:
dummy.head()
Out[35]:
Drop Final Columns¶
Keep only those for classification
In [36]:
final_data = dummy.drop(['weekid' ,'song','performer','long_lasting','top_ten','decade']
,axis=1)
In [37]:
final_data.shape
Out[37]:
In [38]:
#Export final dataset
final_data.to_csv(r'/Users/lcags/Documents/Practicum2/DATA/final_dataset.csv'
,index = False
,header=True)
Resources¶
- https://www.datasciencemadesimple.com/create-frequency-table-of-column-in-pandas-python-2/
- https://www.geeksforgeeks.org/drop-rows-from-the-dataframe-based-on-certain-condition-applied-on-a-column/
- https://www.marsja.se/how-to-use-pandas-get_dummies-to-create-dummy-variables-in-python/
- https://data.world/kcmillersean/billboard-hot-100-1958-2017
In [ ]: